Learning and Understanding User Interface Semantics from Heterogeneous Networks with Multimodal and Positional Attributes

User interfaces (UI) of desktop, web, and mobile applications involve a hierarchy objects (e.g., applications, screens, view class, other types design objects) with multimodal textual visual) positional spatial location, sequence order, level) attributes. We can therefore represent set application UIs as heterogeneous network Such not only represents how users understand the visual layout but also influences would interact through these UIs. To model UI semantics well for different annotation, search, evaluation tasks, this article proposes novel Heterogeneous Attention-based Multimodal Positional (HAMP) graph neural model. HAMP combines networks scaled dot-product attention used in transformers to learn embeddings nodes associated attributes unified manner. is evaluated classification regression tasks conducted on three distinct real-world datasets. Our experiments demonstrate that significantly out-performs state-of-the-art models such tasks. further provide interpretations contribution information understanding relationships between structure prediction we propose Adaptive (AHAMP), which adaptively learns importance edges linking objects. AHAMP’s superior performance over number its ability attributes,